Systems and methods to augment user interaction with the environment outside of a vehicle

ABSTRACT

A method and system for augmented reality. Images of a real world scene are obtained from one or more from one or more image sensors. An orientation and/or location of the image sensors is obtained from one or more state sensors. A real world object at which a predefined pointing object is performing a predefined gesture in the images of the real world scene is identified and data associated with the identified object is displayed on a viewing device. The invention also provides a computer program comprising computer program code means for performing all the steps of the method of the invention when said program is run on a computer.

This application is a continuation of U.S. application Ser. No.15/144,209, filed on May 2, 2016, which is a continuation of U.S.application Ser. No. 14/345,592, filed on Mar. 18, 2014, which is anational stage entry of PCT/IL2012/050376, filed Sep. 19, 2012, whichclaims the priority benefit of U.S. Provisional Application No.61/536,144, filed on Sep. 19, 2011, all of which are incorporated hereinby reference.

PRIOR ART

References considered to be relevant as background to the presentlydisclosed subject matter are listed below:

U.S. Pat. No. 7,126,558;

US Published Patent Application 20110221669:

US Published Patent Application 20110270522;

GB2465280(A);

US Published Patent Application 20120068913:

U.S. Pat. No. 7,215,322;

WO2005/091125;

WO 2010/086866

TECHNOLOGICAL FIELD

The present invention relates to methods and systems for augmentedreality.

Crowley, J. L. et al, linger Tracking as an Input Device for AugmentedReality. Published in the proceedings of the International Workshop onFace and Gesture Recognition, Zurich, Switzerland, June 1995.

Acknowledgement of the above references herein is not to be inferred asmeaning that these are in any way relevant to the patentability of thepresently disclosed subject matter.

BACKGROUND

Augmented reality is a term for a live, direct or an indirect, view of aphysical, real-world environment whose elements are augmented bycomputer-generated information such as text, sound, video, graphics orGPS data. Artificial information about the environment and its objectsis thus overlaid on a real world view or image. Augmentation istypically in real-time and in semantic context with environmentalelements so that information about the surrounding real world of theuser becomes interactive and digitally manipulatable.

The main hardware components for augmented reality are a processor,display, sensors and input devices. These elements, specifically a CPU,display, camera and MEMS sensors such as accelerometer, GPS, or solidstate compass are present in portable device such as smartphones, whichallow them to function as augmented reality platforms.

Augmented reality systems have found applications in entrainment,navigation, assembly processes, maintenance, medical procedures.Portable augmented reality systems have also found applications intourism and sightseeing where augmented reality is used to presentinformation of real world objects and places objects being viewed.

An immersive augmented reality experience is provided using a headmounted display, typically in the form of goggles or a helmet. With ahead-mounted display, virtual visual objects are superimposed on theuser's view of a real world scene. The head mounted display is trackedwith sensors that allow the system to align virtual information with thephysical world. The tracking may be performed, for example, using anyone or more of such technologies as digital cameras or other opticalsensors, accelerometers, GPS, gyroscopes, solid state compasses, RFIDand wireless sensors. Head-mounted displays are either opticalsee-through or video see-through. Optical see-through employs solutionssuch as half-silver mirrors to pass images through the lens and overlayinformation to be reflected into the user's eyes, and transparent LCDprojectors that display the digital information and images directly orindirectly to the user retina.

GENERAL DESCRIPTION

The present invention provides an interactive system for augmentedreality. The interactive system of the invention includes a wearabledata display device that may be incorporated for example, into a pair ofglasses or goggles. The wearable display has a device providing locationextraction capabilities (such as GPS) and a compass. The system alsoincludes a user interface that allows a user to select computergenerated data to augment a real world scene that the user is viewing. Acamera obtains images of the real-world scene being viewed. A processordetects a predefined object in images of the real world scene capturedby the camera such as a user's finger. When the user points to anelement in die scene, data relating to the element are displayed on thedata display device and me superimposed on the user's view of the scene.

Thus, in one of its aspects, the invention provides a method foraugmented reality comprising:

-   -   (a) obtaining images of a real world scene from one or more        image sensors;    -   (b) obtaining front one or more stale sensors one or both of an        orientation and a location data of the image sensors;    -   (c) identifying in the images of the real world scene obtained        by the image sensor or sensors a real world object at which a        predefined pointing object is performing a predefined gesture,        the gesture detection module utilizing data provided by the one        or more state sensors; and    -   (d) presenting data associated with the identified object on a        display of a viewing device.

The image sensor may be selected from a camera a light sensor, an IRsensor, an ultrasonic sensor, a proximity sensor, a CMOS image sensor, ashortwave infrared (SWIR) image sensor or a reflectivity sensor, an IRsensor, an ultrasonic sensor, a proximity sensor, and a reflectivitysensor. One or more of the stale sensors may be selected from an opticalsensor, an accelerometer, GPS, a gyroscope, a compass, magnetic sensor,a sensor indicating the direction of the device relative to the Earth'smagnetic field, a gravity sensor and an RFID detector.

The data associated with the identified object may be obtained bysearching in a memory for data associated with the real world object.

The predefined object may be, for example, a hand, a part of a hand, twohands, parts of two hands, a finger, part of a finger, or a finger tip.

The viewing device may be configured to be worn by a user, for example,glasses or goggles. The viewing device may be incorporated in a mobilecommunication device.

The step of identifying in the images of the real world scene obtainedby the image sensor or sensors may comprise determining a location (X,Y)of the predefined object in an image obtained by the image sensors anddetermining one or both of location and an orientation of the displaydevice provided by the sensors.

The method of the invention may further comprise communicating with anexternal device or website. The communication may comprise sending amessage to an application running on the external device, a servicerunning on the external device, an operating system running on theexternal device, a process running on the external device, one or moreapplications running on a processor of the external device, a softwareprogram running in the background of the external device, or to one ormore services running on the external device. The method may furthercomprise sending a message to an application running on the mobilecommunication device, a service running on the mobile communicationdevice, an operating system running on the mobile communication device,a process running on the mobile communication device, one or moreapplications running on a processor of the mobile communication device,a software program running in the background of the mobile communicationdevice, or to one or more services running on the mobile communicationdevice.

The method may further comprise sending a message requesting a datarelating to a real world object identified in an image front anapplication running on the external device, a service running on theexternal device, an operating system tuning on the external device, aprocess running on the external device, one or more applications runningon a processor of the external device, a software program running in thebackground of the external device, or to one or more services running onthe external device. The method may further comprise sending a messagerequesting a data relating to a real world object identified in an imagefrom an application running on the mobile communication device, aservice running on the mobile communication device, an operating systemrunning on the mobile communication device, a process running on themobile communication device, one or more applications running on aprocessor of the mobile communication device, a software program runningin the background of the mobile communication device, or to one or moreservices running on the mobile communication device.

The message to the external device or website may be a command. Thecommand may be selected from a command to run an application on theexternal device or website, a command to stop an application running onthe external device or website, a command to activate a service runningon the external device or website, a command to slop a service runningon the external device or website, or a command to send data relating toa real world object identified in an image.

The message to the mobile communication device may be a command. Thecommand may be selected from a command to run an application on themobile communication device, a command to stop an application running onthe mobile communication device or website, a command to activate aservice running on the mobile communication device e, a command to stopa service running on the mobile communication device, or a command tosend data relating to a real world object identified in an image.

The method may further comprise receiving from the external device orwebsite data relating to a real world object identified in an image andpresenting the received data to a user.

The communication with the external device or website may be over acommunication network.

The command to the external device may be selected from depressing avirtual key displayed on a display device of the external device;rotating a selection carousel; switching between desktops, running onthe external device a predefined software application; turning off anapplication on the external device; turning speakers on or off; turningvolume up or down: locking the external device, unlocking the externaldevice, skipping to another track in a media player or between IPTVchannels; controlling a navigation application: initiating a call,ending a call, presenting a notification, displaying a notification;navigating in a photo or music album gallery, scrolling web-pages,presenting an email, presenting one or more documents or maps,controlling actions in a game, pointing at a map zooming-in or out on amap or images, painting on an image, grasping an activatable icon andpulling the activatable icon out form the display device, rotating anactivatable icon, emulating touch commands on the external device,performing one or more multi touch commands, a touch gesture command,typing, clicking on a displayed video to pause or play, tagging a frameor capturing a frame from the video, presenting an incoming message;answering an incoming call, silencing or rejecting an incoming call,opening an incoming reminder; presenting a notification received from anetwork community service: presenting a notification generated by theexternal device, opening a predefined application, changing the externaldevice from a locked mode and opening a recent call application,changing the external device from a locked mode and opening an onlineservice application or browser, changing the external device from alocked mode and opening an email application, changing the externaldevice from locked mode and opening an online service application orbrowser, changing the device from a locked mode and opening a calendarapplication, changing the device from a locked mode and opening areminder application, changing the device from a locked mode and openinga predefined application set by a user, set by a manufacturer of theexternal device, or set by a service operator, activating an activatableicon, selecting a menu item, moving a pointer on a display, manipulatinga touch free mouse, an activatable icon on a display, alteringinformation on a display.

In the method of the invention, the predefined gesture may be selectedfrom a swiping motion, a pinching motion of two fingers, pointing, aleft to right gesture, a right to left gesture, an upwards gesture, adownwards gesture, a pushing gesture, opening a clenched fist, opening aclenched fist and moving towards the image sensor, a lapping gesture, awaving gesture, a clapping gesture, a reverse clapping gesture, closinga hand into a fist, a pinching gesture, a reverse pinching gesture, agesture of splaying fingers on a hand, a reverse gesture of splayinglingers on a hand, pointing at an activatable icon, holding anactivating object for a predefined amount of time, clicking on anactivatable icon, double clicking on an activatable icon, clicking fromthe right side on an activatable icon, clicking from the left side on anactivatable icon, clicking from the bottom on an activatable icon,clicking from the top on an activatable icon, grasping an activatableicon the object, gesturing towards an activatable icon the object fromthe right, gesturing towards an activatable icon from the left, passingthrough an activatable icon from the left, pushing the object, clapping,waving over an activatable icon, performing a blast gesture, performinga tapping gesture, performing a clockwise or counter clockwise gestureover an activatable icon, sliding an icon, grasping an activatable iconwith two fingers, and performing a click-drag-release motion.

The data associated with the identified object may be any one or more ofvisual data, audio data, or textual data. The data associated with theidentified object may be an activatable icon. The activatable icon maybe a 2D or 3D activatable icon. The activatable icon may be perceived bya user in a 3D space in front of the user.

The method of the invention may have two or more operational modes. Themethod may change the operational mode of the system upon identificationof a predefined gesture. An operational mode may be specified by any oneor more of the gestures to be identified, algorithms that are active onthe gesture detection module; a resolution of images captured by theimage sensor, and a capture rate of images captured by the image sensor,tire level of details of the data to Ire presented, the activatableicons to be presented to the user, a source of the data to be presented,a level of details of the data to be presented, activatable icons to bedisplayed on the display device, an active on-line service.

The operational mode may be a mode selected from a mode of videorecording of images by the image sensor upon identification of apredefined gesture; a mode of recording sounds by a microphone uponidentification of a predefined gesture and to stop recording uponidentification of another predefined gesture; a mode of continuouslymonitoring video or sound and following a detection of a predefinedgesture, recording the video or sound starting from a predefined amountof time prior to identification of the gesture, and stopping therecording after identification of another predefined gesture, a mode ofadding tags in a captured and real-time recorded video uponidentification of a predefined gesture am mode of selecting an area inthe field of view as captured by the camera, and copying the area toanother location in the field of view and resizing it, a mode employinga tracker on a selected area in an image and is presenting the selectedarea in real-time in the resized and relocated area on the displaydevice, a mode of capturing an image upon identification of a predefinedgesture.

The method of the invention may further comprise running a trackingalgorithm that tracks the identified real world object and maintains thedisplayed associated visual data in a fixed position relative to theidentified real world object.

An object recognition module may be employed to detect the predefinedobject only when the display device has level of motion below apredetermined threshold.

The method may further comprise providing feedback when a predefinedgesture has been identified. The feedback may be, for example, visualfeedback, audio feedback, haptic feedback, directional vibration, airtactile feedback, or ultrasonic feedback. The feedback may be a visualindication in a form selected from an activatable icon displayed on thedisplay device, a change in an activatable icon displayed on the displaydevice, a change in color of an activatable icon displayed on thedisplay device, a change in size of an activatable icon displayed on thedisplay device, animation of an activatable icon displayed on thedisplay device, an indication light, an indicator moving on a displaydevice, an indicator moving on the display device that appears on top ofall other images or video appearing on the display device and theappearance of a glow around the predefined object. The feedback may be avibration, a directional vibration indication, or an air tactileindication.

In the method of the invention, part of an activatable icon displayed onthe display device may not presented where the predefined object islocated, so that the predefined object appears to be on top of theactivatable icon.

Activatable icons may be removed from the display device when thedisplay device has a level of activity above a predefined threshold. Theremoved icons on the display device may be removed, for example, whenthe display device has a level of motion below the predefined threshold.

The method may be brought into an active mode when a predefined actionis performed. The predefined action may be selected from bringing thepredefined object into the field of view from below, when user place thepredefined object in certain location or posse, such as pointing on thebottom right corner of the camera field of view or open his hand in thecamera field of view, perform a predefined gesture such as moving thehand from right to left across the field of view, when an activatableicon is displayed and the user performs a predefined gesture correlateto the activatable icon such as pointing on the activatable icon, orperform a waving gesture in the location where the activatable icon ispresented, or sliding the floating activatable icon from one location tothe other by perform a gesture in the 3D space where the activatableicon is perceived to be located, by touching the device, or tapping onthe device if the device is provided with an accelerometer. As yetanother example, the system may enter the active mode when the userpasses a hand near the device if the device is provided with a proximitysensor, or ultrasonic sensor. The system may also be activated by avoice command, or when the user places the predefined object in aparticular location in the field of view. As yet another example, thesystem may enter the active mode only when there is relevant dataassociated with the real world in the field of view of the user. Whendie system may indicate to the user when there is a relevant data to bepresented, or when it is ready for interaction.

The method of the invention may further comprise attaching a visualindication to a real-world object indicating the existence in a memoryof data correlated with the real-world object. The visual indication maybe overlaid on an image of die real-world object. The visual may lieselected from an activatable icon, a photo, and an image of an envelope.

The method of the invention may further comprise a calibration processto record one or more physical parameters of the predefined object. Thecalibration process may comprise any one or more steps selected frompresenting on the display activatable icons in different locations in a3D space, extracting physical features of the predefined object, anddetermining a correlation between dimensions of the predefined objectand its distance from the camera. The calibration process may comprise astep of constructing a triangle having vertices at one of the imagesensors and at a tip of the predefined object and having a side formedby a user's line of sight. The distance of the real world object fromthe camera may be estimated based on information extracted in thecalibration.

The method may further comprise displaying a keyboard enabling texttyping. The keyboard may be displayed upon detection of a predefinedgesture, such as a gesture from right to left, presenting an open hand,presenting two open hands in a predefined region of the field of view ofan image sensor. The keyboard may be displayed upon performing a clickgesture in a 3D typing area or where a predefined activatable icon isperceived to be located.

The invention also provides a system comprising a device configured toexecute the method of the invention.

The invention also provides a computer program comprising computerprogram code means for performing all the steps of the method of theinvention when said program is run on a computer. The computer programmay be embodied on a computer readable medium.

A user may interact with a visual image typically displayed throughglasses. The user's view of reality is, thus, augmented by theinformation presented on the display. One issue with augmented realitydevices is the manner in which the user interacts with and controls thedevice. Traditional control devices, e.g., a mouse, track ball, or touchscreen, are difficult to use with augmented reality devices. Usinggesture recognition in an augmented reality system is not trivial,because the user, and thus the augmented reality device, is constantlymoving in real time.

The invention thus provides a computer program product containinginstructions for causing a processor to perform a method comprising:

receiving, from an image sensor associated with an augmented reality

device, image information associated with an environment:

displaying, on a display associated with the device, augmentedinformation related to the environment;

recognizing, in the image information, a hand gesture by a user of thedevice;

correlating the hand gesture with the augmented information; and

altering the displayed augmented information based on the correlating.

The augmented information may include at least one of informationassociated with objects in the environment; images associated with theenvironment; and distances associated with the environment.

The correlating may include determining a reference location in threedimensional space of at least a portion of the user's hand, anddetermining in at least one of the augmented information and the imageinformation data associated with the reference location.

The altering may include changing the augmented information as afunction of the data associated with the reference location.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to understand the invention and to see how it maybe carried outin practice, embodiments will now be described, by way of non-limitingexample only, with reference to the accompanying drawings, in which:

FIG. 1 shows schematically a system for augmented reality in accordancewith one embodiment of the invention;

FIG. 2 shows a system for augmented reality comprising a set of gogglesin accordance with one embodiment of the invention;

FIGS. 3a and 3b show diagrams of examples of the system of FIG. 2 inuse;

FIG. 4a show's a view of a real-world scene displayed on a displaydevice of the system of FIG. 2: FIG. 4b shows the view of FIG. 4a withthe user's finger pointing to an object in the view, and FIG. 4c show svisual text relating to the object at which the user's finger ispointing overlaid on the view of FIG. 4 b;

FIG. 5a shows a horn view of a communication device of a system foraugmented reality integral with a communication device in accordancewith another embodiment of the invention. FIG. 5b shows a rear view ofthe communication device of system; and

FIG. 6a shows a diagram of designating an area in the field of view ofan image sensor by the user performing a gesture of “drawing” thecontour of the area. FIG. 6b shows resizing the selected area byperforming a second gesture, FIG. 6c shows the area after resizing, andFIG. 6d shows the area after being dragged to a new location in thefield of view.

DETAILED DESCRIPTION OF EMBODIMENTS

FIG. 1 shows schematically a system 30 for augmented reality inaccordance with one embodiment of the invention. The system 30 includesone or more image sensors 32 configured to obtain images of a real worldscene. Any type of image sensor may be used in the system of theinvention such as a camera alight sensor, an IR sensor, an ultrasonicsensor, a proximity sensor, a CMOS image sensor, a shortwave infrared(SWIR) image sensor or a reflectivity sensor.

The system 30 farther includes a viewing device 34 having one or moredisplay devices 35 that enable a user to see both the real world sceneand external information, such as images, videos, or audio signals,superimposed upon the real world scene. Any type of display device thatallows a user to both see the real world scene and the displayed datamay be used in the system of the invention.

The display devices 35 may comprise, for example, a surface upon whichvisual material is presented to a user or one or more projectors thatdisplay images directly to the user's retina. A processor 36 obtainsorientation and/or location data of the system 30 from one or more stalesensors 38, that may be for example, any one or more of an opticalsensor, an accelerometer, GPS, a gyroscope, a solid slate compasses,magnetic sensor, gravity sensor, and an RFID detector. The processor 36may be for example, a dedicated processor, a general purpose processor,a DSP (digital signaling processor) processor, a GPU (visual processingunit) processor, dedicated hardware, or a processor that can run on anexternal device. The system 30 may run as a software on the viewingdevice 34, or another device 37, such as Smartphone, that incorporatesthe other components of the system 30.

The processor 36 is configured to run a gesture detection module 40 thatidentifies in images of the real world scene obtained by the imagesensor 32 one or more real world objects at which a predefined object ispointing. The real world objects may be, for example, a building or abillboard. Determination of the real world objects utilizes dataprovided by the slate sensors 38. The predefined object may be a user'sfinger or other object such as a stylus or wand.

When the processor 36 has identified a real world object at which thepredefined object is pointing, the processor searches in a memory 42 fordata associated with the identified object. The data may be, forexample, visual data, audio data, or textual data. The visual data maybe textual information relating to the identified object. The processorthen displays the associated visual data associated with the identifiedobject on the display of the viewing device. The memory 42 may beintegral with the system 30 or may be remotely located and accessed overa communication network, such as the Internet. The system 30 may thuscomprise a communication module 39 allowing the system 30 to communicatewith a network, wireless network, cellular network, an external devicesuch as another device 30, a mobile phone, tablet, or an Internetwebsite and so on.

The data may be an activatable icon. As used herein, the term“activatable icon” refers to a region in an image or video associatedwith one or more messages or commands that are activated by a userinteraction. The activatable icons may be for example, a 2D or 3D visualelement such as virtual buttons, a virtual keyboard or icon. Activatableicons are activated by means of one or more predefined objects that arerecognizable by the system, and may be, for example, a stylus, one ormore of a user's hands or a portion of a hand, one or more fingers or aportion of a linger such as a finger tip. Activation of one or more ofthe activatable icons by a predefined object results in the generationof a message or a command addressed to an operating system, one or moreservices, one or more applications, one or more devices, one or moreremote applications, one or more remote services, or one or more remotedevices.

The processor 36 may be configured to send a message or command to thedevice 37 or to a remote device, to an application miming on the device,to a service running on the device 37, and an operating system runningon the device, to a process running on the device, a software programrunning in the background and one or more services running on the deviceor a process running in the device. The message or command may be sentover a communication network such as the Internet or a cellular phonenetwork. The command may be, for example, a command to run anapplication on the device, a command to stop an application running onthe device, a command to activate a service running on the device, acommand to stop a service running on the device, or a command to senddata to the processor 36 relating to a real world object identified inan image by the processor 36.

The command may be a command to the device 37 such as depressing avirtual key displayed on a display device of the device; rotating aselection carousel; switching between desktops, running on the device apredefined software application; turning oil an application on thedevice; turning speakers on or off; turning volume up or down; lockingthe device, unlocking the device, skipping to another track in a mediaplayer or between IPTV channels: controlling a navigation application:initialing a call, ending a call, presenting a notification, displayinga notification; navigating in a photo or music album gallery, scrollingweb-pages presenting an email, presenting one or more documents or maps,controlling actions in a game, controlling interactive video or animatedcontent, editing video or images, pointing at a map, zooming-in or outon a map or images, painting on an image, pushing an activatable iconaway from the display device, grasping an activatable icon anti pullingthe activatable icon out form the display device, rotating anactivatable icon, emulating touch commands on the device, performing oneor more multi-touch commands, a touch gesture command, typing, clickingon a displayed video to pause or play, editing video or music commands,tagging a frame or capturing a frame from the video, cutting a subset ofa video from a video, presenting an incoming message; answering anincoming call, silencing or rejecting an incoming call, opening anincoming reminder; presenting a notification received from a networkcommunity service; presenting a notification generated by the device,changing the device from a locked mode and activating a recent callapplication, changing the device from a locked mode and activating anonline service application or browser, changing the device from a lockedmode and activating an email application, changing the device fromlocked mode and activating an online service application or browser,changing the device from a locked mode and activating a calendarapplication, changing the device from a locked mode and activating areminder application, changing the device from a locked mode andactivating a predefined application set by a user, set by a manufacturerof the device, or set by a service operator, activating an activatableicon, selecting a menu item, moving a pointer on a display, manipulatinga touch free mouse, activating an activatable icon on a display, andaltering information on a display.

The communication module may be used to transmit a message that may beaddressed, for example, to a remote device. The message may be, forexample a command to a remote device. The command may be, for example acommand to run an application on the remote device, a command to slop anapplication running on the remote device, a command to activate aservice running on the remote device, a command to stop a servicerunning on the remote device. The message may be a command to the remotedevice selected from depressing a virtual key displayed on a displaydevice of the remote device; rotating a selection carousel; switchingbetween desktops, running on the remote device a predefined softwareapplication; turning off an application on the remote device; turningspeakers on or off; turning volume up or down; locking the remotedevice, unlocking the remote device, skipping to another track in amedia player or between IPTV channels; controlling a navigationapplication; initiating a call, ending a call, presenting anotification, displaying a notification; navigating in a photo or musicalbum gallery, scrolling web-pages, presenting an email, presenting oneor more documents or maps, controlling actions in a game, pointing at amap, zooming-in or out on a map or images, painting on an image,grasping an activatable icon and pulling the activatable icon out formthe display device, rotating an activatable icon, emulating touchcommands on the remote device, performing one or more multi-touchcommands, a touch gesture command, typing, clicking on a displayed videoto pause or play, tagging a frame or capturing a frame from the video,presenting an incoming message: answering an incoming call, silencing orrejecting an incoming call, opening an incoming reminder; presenting anotification received from a network community service: presenting anotification generated by the remote device, opening a predefinedapplication, changing the remote device from a locked mode and opening arecent call application, changing the remote device from a locked modeand opening an online service application or browser, changing theremote device from a locked mode and opening an email application,changing the remote device from locked mode and opening an onlineservice application or browser, changing the device from a locked modeand opening a calendar application, changing the device from a lockedmode and opening a reminder application, changing the device from alocked mode and opening a predefined application set by a user, set by amanufacturer of the remote device, or set by a service operator,activating an activatable icon, selecting a menu item, moving a pointeron a display, manipulating a touch free mouse, an activatable icon on adisplay, altering information on a display.

The message can be request for data associated with the identifiedobject. The data request message may be addressed to an application, aservice, a process, a thread running on the device, or from anapplication, a service, a process, or a thread running on an externaldevice, or from an online service.

In order to reduce CPU resources, an object recognition module to detectthe predefined object can be employed only when the headset is notmoving significantly as determined from information obtained by thestate sensors.

FIG. 2 shows a system 2 for augmented reality in accordance with oneembodiment of the invention. The system 2 comprises a portable viewingdevice that may be for example, an interactive head-mounted eyepiecesuch as a pair of eyeglasses or goggles 4. The goggles 4 are providedwith a image sensor 6 that obtains images of a real-world scene 8. Thescene 8 may include, for example, one or more buildings 12, or one ormore billboards 14. The goggles may be provided with one or more displaydevices 10 that are located in the goggles 4 so as to be positioned infront of a user's eyes when the goggles 4 are worn by the user. Thedisplay devices 10 may be, for example, see-through devices such astransparent LCD screens through which die real world scene is viewed,together with presenting external data. The system 2 further comprises aprocessor 16 that is configured to identify in images captured by theimage sensors 6, a predefined object performs a gesture or pointing at areal world object in the real world scene 8 or activatable iconsdisplayed to the user. The system 2 also includes one or more locationand/or orientation sensors 23 such as GPS, an accelerometer, agyroscope, a solid state compasses, magnetic sensor, or a gravitysensor.

FIG. 5 shows a system 40 for augmented reality in accordance withanother embodiment of the invention. The system 40 is integrated into amobile communication device 42 such as a mobile phone, tablet, orcamera. A front view of the communication device 42 is shown in FIG. 5a, and a near view of the communication device 42 is shown in FIG. 5b .The communication device 42 is provided with an image sensors 46 on itsrear surface, opposite to the display device, that obtains images of areal-world scene. The communication device 42 is also provided with adisplay device 48 on its front surface that is positioned in front of auser when the camera 46 is directed towards a real world scene. Thedisplay device 48 may be for example, a LCD screen that presents to theuser images of a real world scene obtained by the camera 6, togetherwith visual data, as explained below. The system 40 utilizes the camera46, the display device 48, and the processor of the communication device42, and further comprises one or more slate sensors, contained withinthe housing of the communication device 42 which are not seen in FIG. 5.The processor is configured to identify in images captured by the imagesensors 46 a predefined object pointing at a real world object in thereal world scene.

FIG. 3a shows the system 2 in use. The goggles 4 are placed over theeyes of a user 18. The user faces the real world scene 8 and thus viewsthe scene 8. FIG. 3b shows the system 40 in use. The user 18 holds thecommunication device 42 with the image sensors 46 facing the real worldscene 8 and the display device 48 facing the user.

The system 2 or 40 now executes the following process. The view of thescene 8 that the user would see when using the system 2 or 40 isdisplayed on the display device. FIG. 4a shows the view of the scene 8that the user would see when using the system 2 or 40 to view the realworld scene 8. The processor 36 analyzes images obtained by the imagesensors to determine when a predefined object in images captured by theimage sensors is performing a predefined gesture in relation to a realworld object in the real world scene 8.

The viewing device 34, such as the goggles 4 or the communication device42 is often not stable in use, due to movement of the user as occursduring walking, or movement of the user's head or hand. In thissituation, the signal generated by the sensors 38 may be noisy andinaccurate. In this case, the machine vision module 37 runs a trackingalgorithm that tracks the identified real world object and maintains thedisplayed associated visual data in a fixed position relative to theidentified real world object.

The predefined gesture relating to a real world object or to anactivatable icon, may be, for example, pointing at the real world objector an activatable icon, or performing a swiping gesture over the realworld object or an activatable icon. The activatable icon may or may notbe correlated to a real world object.

Other possible predefined gestures include a swiping motion, a pinchingmotion of two fingers such as with the fore finger and thumb or themiddle linger and thumb, pointing, a left to right gesture, a right toleft gesture, an upwards gesture, a downwards gesture, a pushinggesture, opening a clenched fist, opening a clenched fist and movingtowards the image sensor, a tapping gesture, a waving gesture, aclapping gesture, a reverse clapping gesture, closing a hand into afist, a pinching gesture, a reverse pinching gesture, a gesture ofsplaying fingers on a hand, a reverse gesture of splaying fingers on ahand, pointing at an activatable icon, or at a real world object,pointing at an activatable icon or a real world object for a predefinedamount of time, clicking on an activatable icon or real world object,double clicking on an activatable icon or real world object, clickingwith a forefinger on an activatable icon or real world object, clickingwith the middle linger on an activatable icon or real world object,clicking from the bottom on an activatable icon or real world object,clicking from the top on an activatable icon, grasping an activatableicon or real world object gesturing towards an activatable icon or realworld object from the right, gesturing towards an activatable icon orreal world object from the left, passing through an activatable icon orreal world object from the left, pushing the activatable icon or realworld object, clapping or waving over an activatable icon or real worldobject, performing a blast gesture, performing a tapping gesture,performing a clockwise or counter clockwise gesture over an activatableicon or real world object, sliding an activatable icon or real worldobject, grasping an activatable icon or real world object with twofingers, or performing a click-drag-release motion.

The predefined object may be, for example, a user hand, a part of auser's hand, such as the user's linger 20 or parts of two differenthands. Alternatively, the predefined object may be a stylus or wand.

When the processor 16 determines that a predefined gesture has beenperformed, this may be indicated to the user by any type of feedback,such as visual feedback, audio feedback, haptic feedback, directionalvibration, air tactile feedback, or ultrasonic feedback. The feedbackmay be a visual indication in a form selected from an activatable icondisplayed on a display device, a change in an activatable icon on adisplay device, a change in color of an activatable icon on a displaydevice, a change in size of an activatable icon, animation of anactivatable icon, an indication light, an indicator moving on a displaydevice, a vibration, a directional vibration indication, an air tactileindication. The indication may be provided by an indicator moving on adisplay device that appears on top of all other images or videoappearing on the display device. Visual feedback may be the appearanceof a glow around the predefined object when a system recognizes thepredefined object.

The gesture detection module 40 may use any method for detecting thepredefined objects in images obtained by the image sensor 32. Forexample, the gesture detection module may detect die predefined objectas disclosed in WO2005/091125 or WO 2010/086866.

The processor 16 is further configured to determine the real worldobject in the scene 8 towards which the predefined gesture wasperformed. Thus, for example, in the image shown in FIG. 4b , theprocessor 16 would determine that the user's linger 20 is pointing atthe billboard 14 by determining the fingertip location (X,Y) in dieimage and combining this information with the location of the user andthe orientation of the goggles 4 from the state sensors 21. The realworld object is thus identified by the processor without presenting tothe user a cursor or other marker to indicate the real world object thatthe user wishes to select, enabling a direct pointing on a real worldobject to start an interaction. The processor 16 searches in a memory,which may be integral with the processor 16 or may be remotely located,for data relating to the real-world object to which the user's finger 20is pointing. For example, the memory may have stored data relating tothe billboard 14. When the user points to an object in the scene 8 whosedata is stored in the memory or is extracted from a remote server suchas an Internet site, the data is displayed on the display device 10superimposed on the user's view of the scene. Thus, when the user pointsto the billboard 14 (FIG. 3), visual data 21 relating to the billboard14 is displayed on the display device 10, as shown in FIG. 4 c.

The visual data 21 may be static or animated. The visual data 21 mayinclude one or more an activatable icons, such that when a predefinedgesture is performed relative to one of the activatable icons, a commandassociated with the activatable icon is executed. The command may be,for example, to display specific visual material relating to theselected real world object. The activatable icons may be 2D or 3Dactivatable icons and may be presented to the user so that the userperceives the icon in front of him in a 3D space. As used herein, anactivatable icon is a region in a 2D or 3D image or video associatedwith one or more messages activated by user interaction. The activatableicons may be, for example, a 2D or 3D visual element. The activatableicons may be virtual buttons, a virtual keyboard, a 2D or 3D activatableicon, a region in an image or a video. An activatable icon may consistof two or more activatable icons. The processor may not present pan ofthe activatable icon where the predefined object is located, so dial thepredefined object appears to be on top of the activatable icon. Theactivatable icons may be removed when the user rapidly moves his headand then returned when the head motion is below a predefined motionspeed.

The system 2 may have two or more operational modes and the processor 16may be configured to identify one or more predefined gestures to changebetween the operational modes. Thus, a gesture may be used to turn diesystem on or off, select the source of the visual material to bepresented, select the level of details of the visual material to bepresented, select the buttons or activatable icons to be presented tothe user, or activate an online service, such as an online servicerelated to a selected real world object. Yet another mode of operationmay be to start video recording of images by the image sensor and/orrecording of sounds by a microphone upon identification of a predefinedgesture and to slop recording upon identification of another predefinedgesture. Yet another mode of operation is continuously monitoring videoand/or sound, hut following a detection of a predefined gesture, thevideo/sound is recorded starting from a predetermined amount of timeprior to identification of the gesture, and stopping the recording afteridentification of another predefined gesture. The predetermined time maybe defined by the user. Yet another mode of operation is adding tags ina captured and real-time recorded video upon identification of apredefined gesture.

Yet another mode of operation is shown in FIG. 6. In FIG. 6a , an area62 in the field of view 60 as captured by the image sensor is designatedby the user performing a gesture of “drawing” the contour of the area,shown by phantom lines in FIG. 6. The selected area is then resized bythe user performing a second gesture, such as separating two fingers orbringing two fingers closer together as indicated by the arrows 66 inFIG. 6b , until the selected area attains the desired size (67 in FIG.6c ). The area 67 is then drugged to a new location in the field of view(FIG. 6d ) and copied in the new location in the field of view. Thesystem then employs a tracker on the selected area and the selected areais presented in real-time in the resized and relocated area set by theuser on the display device.

In order to minimize CPU resources, for each displayed activatable icon,a region of images containing a displayed activatable icon hounding boxaround a displayed activatable icon may be defined that remains fixed.The system employs a machine vision tracker to track this bounding box.The distance between the locations of the hounding boxes in two framesof a video sequence is less than a predefined distance, as determinedusing a video tracker, and the correlation value of the tracker of thebounding box is below a predefined value.

When the system is in an operational mode in which only activatableicons may be activated, and real world objects cannot be activated, CPUcan be minimized by searching for the predefined object only in thevicinity of each displayed activatable icon. In order to reduce CPU evenfurther, the objection recognition module is not activated all tin: timebut only when the headset is not moving significantly as determined frominformation obtained by a stale sensors.

A user may choose different filters to screen data correlated withreal-world objects, such as a filter “display data generated only byfriends”, or display data from registered sources, or data generated inthe last three months.

The system 2 may have a stand-by mode in which the power consumption bythe system 2 is minimal. The active mode may be different from thestand-by mode, for example, in the number of video frames per secondthat are being analyzed by the system, the resolution of images that arebeing analyzed, the portion of the image frame that is being analyzed,and/or the detection modules that are activated. The system 2 can bebrought to the active mode by any technique. For example, the system 2may be brought to the active mode by bringing the predefined object intothe field of view from below, when user place the predefined object incertain location or posse, such as pointing on the bottom right cornerof the camera field of view or open his hand in the camera field ofview, perform a predefined gesture such as moving the hand from right toleft across the field of view, when an activatable icon is displayed andthe user performs a predefined gesture correlate to the activatable iconsuch as pointing on the activatable icon, or perform a waving gesture inthe location where the activatable icon is presented, or sliding thefloating activatable icon from one location to the other by perform agesture in the 3D space where the activatable icon is perceived to belocated, by touching the device, or tapping on the device if the deviceis provided with an accelerometer. As yet another example, the systemmay enter the active mode when the user passes a hand near the device ifthe device is provided with a proximity sensor, or ultrasonic sensor.The system may also be activated by a voice command, or when the userplaces the predefined object in a particular location in the field ofview. As yet another example, the system may enter the active mode onlywhen there is relevant data associated with the real world in the fieldof view of the user. When the system may indicate to the user when thereis a relevant data to be presented, or when it is ready for interaction.

A visual indication may be attached to a real-world object to let theuser know that there is data correlated with the real-world object.

Indication of relevant data may be overlaid on the location of thereal-world object as a small visual indication such as an activatableicon of “i” may indicate information, and a logo of “photos” mayindicate images related to the real-world object, or a logo of “envelop”indicates a message that was left by a friend or other user correlatedto the real-world object. When the user performs a predefined gesturecorrelated to the activatable icon, the data may be presented.

The system 2 may be configured to undergo a calibration process torecord various physical parameters of the predefined object so as tofacilitate identification of the predefined object in images obtained bythe camera by the processor 2. This may be done, for example, bypresenting to the user on the display activatable icons in differentlocations in the 3D space, and extracting physical features of thepredefined object such as its size or orientation of the predefinedobject, and determining a correlation between the dimensions of thepredefined object and its distance from the camera. The calibration mayinvolve calculating the triangular of camera, die user's line of sightand the tip of the predefined object to determine die user is pointingat. The accuracy is improved by estimating the distance of the realworld object from the camera based on information extracted in thecalibration.

The processor may be configured to identify in images obtained by thecamera of the real world scene by another user of the system of theinvention. The identification of another user in the real world scenemay be performed, for example, by informing a remote server of thelocations of the devices in a particular geographical area. Thelocations of the other devices can be sent to all of the devices in thegeographical area.

When a communication link exists between two systems of the invention,the two systems may be used for game playing. The other user may berepresented to as an avatar with whom the user can interact by gesturessuch as send a message to the other user such as “like”.

The processor may be configured to display a keyboard that enables texttyping with one or more fingers or hands. Display of the keyboard may beinitiated upon detection of a predefined gesture such as a gesture fromright to left, or by the using presenting an open hand, or two openhands in a predefined region of the field of view of the camera, such asthe bottom part of the field of view. Yet another way to initiate thedisplay of the keyboard is when the user performs a click gesture in the3D space where the typing area or an activatable icon is perceived to belocated. The keyboard may be used, for example, in order to, write anote, conduct a search or to communicate with online services (such asSkype or twiner) by typing on virtual keyboard. The system may notpresent part of the keyboard where the predefined object is located, sothat the predefined object appears to be on top of the keyboard tocreate the illusion that the predefined object such as a user's handappears to be “over” the keyboard.

When the system is in a typing mode, an animated hand may be presentedon the keyboard whose position is correlated with the user's hands andfingers. The fingertips of the animated hands may be located above avirtual keystroke at the location where the character of the keystrokeis seen. The keyboard and the animated hands are preferably opaque, sothat the user is unable see the background behind the keyboard. Thislends to make the keyboard clearer to the user.

The invention claimed is:
 1. A device, comprising: at least one processor configured to: receive, from an image sensor, video frames of image information associated with a real world scene; detect, in the image information, a predefined hand gesture performed by a user; identify in the image information one or more real-world objects other than the user's hand and displayed information, wherein identifying the one or more real-world objects is associated with the detected predefined hand gesture; designate selected image information associated with the identified one or more real-world objects in the video frames, wherein the selected image information is associated with the real world scene, wherein information associated with the real world scene does not include the user's hand and the displayed information; tag the selected image information associated with the identified one or more real-world objects other than the user's hand, at least in part, on the detected gesture; designate an area associated with the selected image information; and record at least one of video information associated with the designated area of the selected image information and with a time prior to the detected predefined hand gesture.
 2. The device of claim 1, wherein the predefined hand gesture includes at least one of: drawing a contour associated with real-world objects or, pointing at real-world object.
 3. The device of claim 1, wherein the at least one processor is further configured to resize the designated area.
 4. The device of claim 1, wherein the recording starts a predefined time prior to the detected hand gesture.
 5. The device of claim 1, wherein the recording stops a predefined time after the detected hand gesture.
 6. The device of claim 1, wherein the at least one processor is further configured to stop the recording based on a detection of a second gesture.
 7. The device of claim 1, wherein the at least one processor is further configured to capture a frame from the video frames of image information in association with a second predefined hand gesture detected in the image information.
 8. The device of claim 1, wherein the time is a predefined amount of time defined by the user.
 9. The device of claim 1, wherein the at least processor is further configured to run a tracking algorithm that tracks an identified real world object in the image information, in response to the detected second predefined hand gesture.
 10. The device of claim 1, wherein the at least one processor is further configured to employ a tracker on the selected image information to present the real-time view of the real-world object in the selected information in a resized area in the field of view of the user.
 11. A method of operating a device, the method comprising: receiving, by at least one processor from an image sensor, video frames of image information associated with a real world scene; detecting, by the at least one processor in the image information, a predefined hand gesture performed by a user; designating, in the video frames, by the at least one processor, an area of selected image information associated with a real-world object other than the user's hand and displayed information, wherein the selected image information is associated with the real world scene that does not include the user's hand and the displayed information, the designated area being associated with the detected predefined hand gesture; tracking the selected image information associated with the real world scene not including the user's hand; based on the tracking, designating an area of selected image information; and recording at least one of video information associated with the designated area of the selected image information and with a time associated with the detected predefined hand gesture.
 12. The method of claim 11, wherein the predefined hand gesture includes at least one of: drawing a contour of the designated area, pointing at real-world object.
 13. The method of claim 11, wherein the at least one processor is further configured to resize the designated area.
 14. The method of claim 13, wherein the second predefined hand gesture comprises separating two fingers, or bringing two fingers closer together.
 15. The method of claim 11, further comprising zooming in or out in association with a second predefined hand gesture detected in the image information.
 16. The method of claim 11, further comprising employing a tracker on the selected area to present the real-time view of the real-world object in the selected information in a resized area in the field of view of the user.
 17. A non-transitory computer-readable medium storing instructions that, when executed, cause at least one processor to perform a method of operating a device, the method comprising: receiving, from an image sensor, video frames of image information associated with a real world scene; detecting, in the image information, a predefined area selected hand gesture performed by a user; designating an area of selected image information associated with the detected hand gesture, in the video frames, wherein the selected image information is associated with the real world scene that does not include the user's hand and displayed information; and tracking the selected image information associated with the real world scene that does not include the user's hand to record a real-time video of selected image information associated with the real world scene in the selected information.
 18. The non-transitory computer-readable medium of claim 17, wherein the real world scene includes, at least in part, information displayed on a display device. 